Performance Of Echo State Network Research Articles

Impact of time-history terms on reservoir dynamics and prediction accuracy in echo state networks

The echo state network (ESN) is an excellent machine learning model for processing time-series data. This model, utilising the response of a recurrent neural network, called a reservoir, to input signals, achieves high training efficiency. Introducing time-history terms into the neuron model of the reservoir is known to improve the time-series prediction performance of ESN, yet the reasons for this improvement have not been quantitatively explained in terms of reservoir dynamics characteristics. Therefore, we hypothesised that the performance enhancement brought about by time-history terms could be explained by delay capacity, a recently proposed metric for assessing the memory performance of reservoirs. To test this hypothesis, we conducted comparative experiments using ESN models with time-history terms, namely leaky integrator ESNs (LI-ESN) and chaotic echo state networks (ChESN). The results suggest that compared with ESNs without time-history terms, the reservoir dynamics of LI-ESN and ChESN can maintain diversity and stability while possessing higher delay capacity, leading to their superior performance. Explaining ESN performance through dynamical metrics are crucial for evaluating the numerous ESN architectures recently proposed from a general perspective and for the development of more sophisticated architectures, and this study contributes to such efforts.

The connectivity degree controls the difficulty in reservoir design of random boolean networks.

Reservoir Computing (RC) is a paradigm in artificial intelligence where a recurrent neural network (RNN) is used to process temporal data, leveraging the inherent dynamical properties of the reservoir to perform complex computations. In the realm of RC, the excitatory-inhibitory balance b has been shown to be pivotal for driving the dynamics and performance of Echo State Networks (ESN) and, more recently, Random Boolean Network (RBN). However, the relationship between b and other parameters of the network is still poorly understood. This article explores how the interplay of the balance b, the connectivity degree K (i.e., the number of synapses per neuron) and the size of the network (i.e., the number of neurons N) influences the dynamics and performance (memory and prediction) of an RBN reservoir. Our findings reveal that K and b are strongly tied in optimal reservoirs. Reservoirs with high K have two optimal balances, one for globally inhibitory networks (b < 0), and the other one for excitatory networks (b > 0). Both show asymmetric performances about a zero balance. In contrast, for moderate K, the optimal value being K = 4, best reservoirs are obtained when excitation and inhibition almost, but not exactly, balance each other. For almost all K, the influence of the size is such that increasing N leads to better performance, even with very large values of N. Our investigation provides clear directions to generate optimal reservoirs or reservoirs with constraints on size or connectivity.

Time Series Prediction of ESN Based on Chebyshev Mapping and Strongly Connected Topology

This paper introduces a novel approach called Chebyshev mapping and strongly connected topology for optimization of echo state network (ESN). To enhance the predictive performance of ESNs for time series data, Chebyshev mapping is employed to optimize the irregular input weight matrix. And the reservoir of the ESN is also replaced using an adjacency matrix derived from a digital chaotic system, resulting in a reservoir with strong connectivity properties. Numerical experiments are conducted on various time series datasets, including the Mackey–Glass time series, Lorenz time series and solar sunspot numbers, validating the effectiveness of the proposed optimization methods. Compared with the traditional ESNs, the optimization method proposed in this paper has higher predictive performance, and effectively reduce the reservoir’s size and model complexity.

Robust echo state network with Cauchy loss function and hybrid regularization for noisy time series prediction

Noisy time series prediction is a hot research topic in practical applications. Echo state networks (ESNs) have superior performance on time series prediction. However, the ill-posed problem may weaken the generalization performance of ESNs when dealing with real data, because of the noise and the overlarge reservoir. To solve this problem, this paper proposes a robust ESN with a Cauchy loss function and hybrid regularizations for noisy time series prediction. First, the Cauchy loss function is used to suppress the large noise mixed in the real data. Then, an improved L1 regularization and the L2 regularization are combined to shrink the larger output weights and produce the sparse solution. Particularly, a simple iterative algorithm is proposed to calculate the output weights of the network. Finally, simulation results on both synthetic and real-world data sets show the super performance of the proposed model.

A combination forecasting model of wind speed based on decomposition

Due to the intermittent, fluctuating and random characteristics of wind system, the output of wind power will become unstable with the change of wind, which brings severe challenges to the safe and stable operation of the power system. An effective way to solve this problem is to accurately forecast the wind speed. This paper presents a novel wind speed combination forecasting model based on decomposition. The innovation of the forecasting model is as follows. (a) In view of the unstable characteristics of wind speed, variational mode decomposition algorithm is introduced to decompose the historical wind speed data to obtain a series of stable components with different frequencies. (b) Echo state network with good forecasting ability is selected as the forecasting model of each component. (c) To solve the problem that the forecasting performance of echo state network is greatly affected by the parameters of the reservoir, an improved whale optimization algorithm is proposed to optimize these parameters. The optimized echo state network improves the forecasting effect. (d) The final forecasting results are obtained by adding the forecasting values of each component. (e) The performance of the developed forecasting model is verified by using two actual collected data sets of ultra-short-term wind speed and short-term wind speed. Compared with some state-of-the-art forecasting models, the comparison result curve between the forecasting value and actual value of wind speed, the forecasting error distribution, the histogram of the forecasting error distribution, the performance indicators, related statistical indicators, and Taylor diagram show that the developed forecasting model has higher prediction accuracy and is able to reflect the change laws of wind speed correctly.

Network traffic prediction method based on echo state network with adaptive reservoir

AbstractNetwork traffic prediction is of great significance to resource management in cyber‐physical systems (CPSs). In particular, network traffic is a nonlinear time series. Echo state network (ESN) is a new neural network with strong nonlinear processing capacity and short‐term memory capacity, and thus can achieve good performance in predicting nonlinear time series. However, network traffic has various characteristics such as self‐similarity, chaos, mutability. As the core of ESN, the reservoir will be fixed rather than adjustable once it is generated, which limits the prediction performance of ESN in different network traffic. To achieve universal excellent prediction performance, this paper proposes a new network traffic prediction method based on ESN with adaptive reservoir (ESN‐AR). First, the framework of ESN‐AR is constructed for network traffic prediction, in which the idea of generative adversarial network (GAN) is incorporated into ESN to adaptively adjust the reservoir. Specifically, ESN is used as the generative model to predict network traffic and feedforward neural network (FNN) is used as the discriminative model to distinguish between the real network traffic and the predicted network traffic. Second, the adversarial training algorithm of ESN‐AR is proposed to obtain the appropriate reservoir depending on the network traffic characteristics. Finally, ESN‐AR is applied to the prediction of three actual network traffic with different characteristics. Simulation results show that compared with the state‐of‐the‐art models, the proposed method achieves more accurate and stable prediction performance.

Robust speed prediction of high-speed trains based on improved echo state networks

The accurate and robust speed prediction of high-speed trains (HSTs) is a challenging task in the automatic train operation (ATO) because HSTs operate in an open-air situation with much noise and many uncertainties. This paper contributes to the development of robust speed prediction methods for the ATO of HSTs based on improved echo state networks (ESNs). Firstly, an adaptive temporal scale selection approach is introduced to improve the accuracy and efficiency of ESN modeling, due to the importance of proper temporal scale selection in relation to the sound prediction performance of ESNs. Also, a random weight scaling mechanism is employed to enhance the feasibility and robustness of the proposed method, as the learning ability of ESNs lies in the constrained random connection weights. Furthermore, several conditions for the stability of the closed-loop control system are given. Our experiment results demonstrate that the proposed method successfully achieves sound performance in terms of speed prediction accuracy, efficiency and robustness.

Bayesian Optimized Echo State Network Applied to Short-Term Load Forecasting

Load forecasting impacts directly financial returns and information in electrical systems planning. A promising approach to load forecasting is the Echo State Network (ESN), a recurrent neural network for the processing of temporal dependencies. The low computational cost and powerful performance of ESN make it widely used in a range of applications including forecasting tasks and nonlinear modeling. This paper presents a Bayesian optimization algorithm (BOA) of ESN hyperparameters in load forecasting with its main contributions including helping the selection of optimization algorithms for tuning ESN to solve real-world forecasting problems, as well as the evaluation of the performance of Bayesian optimization with different acquisition function settings. For this purpose, the ESN hyperparameters were set as variables to be optimized. Then, the adopted BOA employs a probabilist model using Gaussian process to find the best set of ESN hyperparameters using three different options of acquisition function and a surrogate utility function. Finally, the optimized hyperparameters are used by the ESN for predictions. Two datasets have been used to test the effectiveness of the proposed forecasting ESN model using BOA approaches, one from Poland and another from Brazil. The results of optimization statistics, convergence curves, execution time profile, and the hyperparameters’ best solution frequencies indicate that each problem requires a different setting for the BOA. Simulation results are promising in terms of short-term load forecasting quality and low error predictions may be achieved, given the correct options settings are used. Furthermore, since there is not an optimal global optimization solution known for real-world problems, correlations among certain values of hyperparameters are useful to guide the selection of such a solution.

Novel soft sensor development using echo state network integrated with singular value decomposition: Application to complex chemical processes

It is of great importance to develop advanced soft sensors for ensuring the safety and stability of complex industrial processes. Unluckily, with the increasing scale of chemical processes, it becomes more and more demanding to develop soft sensor with high accuracy. In addition, most of industrial processes are dynamic. As a result, the soft sensors developed using static models cannot achieve acceptable performance. In order to handle this problem, the Echo state network (ESN) as a kind of recurrent neural network is selected. However, the output weights of ESN are calculated linearly. On one hand, the collinear in the reserve layer outputs may decrease the performance; on the other hand, the over-fitting problem may occur. To enhance and improve the ESN performance, singular value decomposition based ESN (SVD-ESN) is presented. In the SVD-ESN method, the singular value decomposition instead of the traditional least square is adopted to calculate the weights between the output layer and the reserve layer. Through singular value analysis in the outputs of the reserve layer, appropriate defining parameters are selected to enhance the accuracy and ensure the computing speed. As a result, the collinearity and over-fitting problem is solved; then the performance of ESN is enhanced. To test and validate the performance of SVD-ESN, the proposed SVD-ESN is developed as soft sensor for the High Density Polyethylene (HDPE) production process and Purified Terephthalic Acid (PTA) production process. Compared with the conventional ESN, Extreme Learning Machine (ELM), Dynamic Window based ELM (DW-ELM) and Long Short-Term Memory (LSTM), the simulation results show that the proposed SVD-ESN model obtains better performance in terms of prediction accuracy, which conforms that the proposed SVD-ESN can be used as an effective dynamic model for developing accurate soft sensors.

Nonlinear process modelling using echo state networks optimised by covariance matrix adaption evolutionary strategy

Echo state networks (ESN) have been shown to be an effective alternative to conventional recurrent neural networks (RNNs) due to its fast training process and good performance in dynamic system modelling. However, the performance of ESN can be affected by the randomly generated reservoir. This paper presents nonlinear process modelling using ESN optimized by covariance matrix adaption evolutionary strategy (CMA-ES). CMA-ES is used to optimize the structural parameters of ESN: reservoir size, spectral radius, and leak rate. The proposed method is applied to three case studies: modelling a time series, modelling a conic tank, and modelling a fed-batch penicillin fermentation process. The results are compared with those from the original ESN, long short-term memory network, GA-ESN (ESN optimized by genetic algorithm), and feedforward neural networks. It is shown that the proposed method gives much better performance than the original ESN and other networks on all the three case studies.

An echo state network architecture based on quantum logic gate and its optimization

Quantum neural network (QNN) is developed based on two classical theories of quantum computation and artificial neural networks. It has been proved that quantum computing is an important candidate for improving the performance of traditional neural networks. In this work, inspired by the QNN, the quantum computation method is combined with the echo state networks (ESNs), and a hybrid model namely quantum echo state network (QESN) is proposed. Firstly, the input training data is converted to quantum state, and the internal neurons in the dynamic reservoir of ESN are replaced by qubit neurons. Then in order to maintain the stability of QESN, the particle swarm optimization (PSO) is applied to the model for the parameter optimizations. The synthetic time series and real financial application datasets (Standard & Poor's 500 index and foreign exchange) are used for performance evaluations, where the ESN, autoregressive integrated moving average (ARIMAX) are used as the benchmarks. Results show that the proposed PSO-QESN model achieves a good performance for the time series predication tasks and is better than the benchmarking algorithms. Thus, it is feasible to apply quantum computing to the ESN model, which provides a novel method to improve the ESN performance.

Evolving Local Plasticity Rules for Synergistic Learning in Echo State Networks.

Existing synaptic plasticity rules for optimizing the connections between neurons within the reservoir of echo state networks (ESNs) remain to be global in that the same type of plasticity rule with the same parameters is applied to all neurons. However, this is biologically implausible and practically inflexible for learning the structures in the input signals, thereby limiting the learning performance of ESNs. In this paper, we propose to use local plasticity rules that allow different neurons to use different types of plasticity rules and different parameters, which are achieved by optimizing the parameters of the local plasticity rules using the evolution strategy (ES) with covariance matrix adaptation (CMA-ES). We show that evolving neural plasticity will result in a synergistic learning of different plasticity rules, which plays an important role in improving the learning performance. Meanwhile, we show that the local plasticity rules can effectively alleviate synaptic interferences in learning the structure in sensory inputs. The proposed local plasticity rules are compared with a number of the state-of-the-art ESN models and the canonical ESN using a global plasticity rule on a set of widely used prediction and classification benchmark problems to demonstrate its competitive learning performance.

Effects of singular value spectrum on the performance of echo state network

Reservoir computing (RC), as an efficient and simple paradigm of training recurrent networks, has attracted considerable attention for its superior performance on many tasks. Echo state network (ESN) is one of typical representatives of RC, whose good performance mainly depends on the selection of right parameters. Hence, it is necessary for users to understand the inter-relationship between the performance and the parameters of ESN. Using computational experiments and statistical analysis, this paper systematically analyzes the effects of singular value spectrum on the performance of ESN, such as memory capacity, robustness, generalization performance, dynamics and so on. To simultaneously control the singular value spectrum and the sparsity when designing an ESN, two algorithms are introduced to construct the reservoir with the predefined singular value spectrum and sparsity, and their benefits are assessed on some benchmark problems.

Research on Sparsity of Output Synapses in Echo State Networks

This paper presents an improved model of echo state networks (ESNs) and gives the definitions of energy consumption, energy efficiency, etc. We verify the existence of redundant output synaptic connections by numerical simulations. We investigate the relationships among energy consumption, prediction step, and the sparsity of ESN. At the same time, the energy efficiency and the prediction steps are found to present the same variation trend when silencing different synapses. Thus, we propose a computationally efficient method to locate redundant output synapses based on energy efficiency of ESN. We find that the neuron states of redundant synapses can be linearly represented by the states of other neurons. We investigate the contributions of redundant and core output synapses to the performance of network prediction. For the prediction task of chaotic time series, the predictive performance of ESN is improved about hundreds of steps by silencing redundant synapses.

The combination of circle topology and leaky integrator neurons remarkably improves the performance of echo state network on time series prediction.

Recently, echo state network (ESN) has attracted a great deal of attention due to its high accuracy and efficient learning performance. Compared with the traditional random structure and classical sigmoid units, simple circle topology and leaky integrator neurons have more advantages on reservoir computing of ESN. In this paper, we propose a new model of ESN with both circle reservoir structure and leaky integrator units. By comparing the prediction capability on Mackey-Glass chaotic time series of four ESN models: classical ESN, circle ESN, traditional leaky integrator ESN, circle leaky integrator ESN, we find that our circle leaky integrator ESN shows significantly better performance than other ESNs with roughly 2 orders of magnitude reduction of the predictive error. Moreover, this model has stronger ability to approximate nonlinear dynamics and resist noise than conventional ESN and ESN with only simple circle structure or leaky integrator neurons. Our results show that the combination of circle topology and leaky integrator neurons can remarkably increase dynamical diversity and meanwhile decrease the correlation of reservoir states, which contribute to the significant improvement of computational performance of Echo state network on time series prediction.

Echo State versus Wavelet Neural Networks: Comparison and Application to Nonlinear System Identification

This paper examines two popular neural networks that have been successfully utilized in a wide variety of applications: echo state networks (ESN) and wavelet neural networks (WNN). It introduces innovations in the structure of ESN that result in major improvements in their performance. By adding a feedforward component to the networks, and by tuning their weights using recursive least squares, we can drastically reduce the number of hidden layer neurons while reducing their error. We demonstrate the improvement in the performance of ESN and compare their performance to that of WNN by application to the identification of an un-manned vehicle. Different ESN and WNN are utilized to identify the model of the vehicle and examine the effects of adding a feedforward term to ESN and WNN. Simulation results shows that adding a feedforward term to ESN and WNN can reduce the number of hidden nodes drastically while reducing the identification error of ESN. Because the number of internal units in ESN can be very high for nonlinear systems, adding this term makes ESN more suitable for online identification and control applications. Our results show that, even though both networks yield acceptable performance, ESN outperform WNN in terms of accuracy but have a slightly higher computational cost.

Multiplex visibility graphs to investigate recurrent neural network dynamics

A recurrent neural network (RNN) is a universal approximator of dynamical systems, whose performance often depends on sensitive hyperparameters. Tuning them properly may be difficult and, typically, based on a trial-and-error approach. In this work, we adopt a graph-based framework to interpret and characterize internal dynamics of a class of RNNs called echo state networks (ESNs). We design principled unsupervised methods to derive hyperparameters configurations yielding maximal ESN performance, expressed in terms of prediction error and memory capacity. In particular, we propose to model time series generated by each neuron activations with a horizontal visibility graph, whose topological properties have been shown to be related to the underlying system dynamics. Successively, horizontal visibility graphs associated with all neurons become layers of a larger structure called a multiplex. We show that topological properties of such a multiplex reflect important features of ESN dynamics that can be used to guide the tuning of its hyperparamers. Results obtained on several benchmarks and a real-world dataset of telephone call data records show the effectiveness of the proposed methods.

Determination of the Edge of Criticality in Echo State Networks Through Fisher Information Maximization.

It is a widely accepted fact that the computational capability of recurrent neural networks (RNNs) is maximized on the so-called "edge of criticality." Once the network operates in this configuration, it performs efficiently on a specific application both in terms of: 1) low prediction error and 2) high short-term memory capacity. Since the behavior of recurrent networks is strongly influenced by the particular input signal driving the dynamics, a universal, application-independent method for determining the edge of criticality is still missing. In this paper, we aim at addressing this issue by proposing a theoretically motivated, unsupervised method based on Fisher information for determining the edge of criticality in RNNs. It is proved that Fisher information is maximized for (finite-size) systems operating in such critical regions. However, Fisher information is notoriously difficult to compute and requires the analytic form of the probability density function ruling the system behavior. This paper takes advantage of a recently developed nonparametric estimator of the Fisher information matrix and provides a method to determine the critical region of echo state networks (ESNs), a particular class of recurrent networks. The considered control parameters, which indirectly affect the ESN performance, are explored to identify those configurations lying on the edge of criticality and, as such, maximizing Fisher information and computational performance. Experimental results on benchmarks and real-world data demonstrate the effectiveness of the proposed method.

Optimizing the echo state network based on mutual information for modeling fed-batch bioprocesses

Echo state networks (ESNs) have become one of the most effective dynamic neural networks because of its excellent fitting performance in real-valued time series modeling tasks and simple training processes. The original ESN concept used randomly fixed created reservoirs, and this concept is considered to be one of its main advantages. However, ESNs have been criticized for its randomly created connectivity and weight parameters. Determining the appropriate weight parameters for a given task is an important problem. An optimization method based on mutual information (MI) is proposed in this study to optimize the input scaling parameters and the structure of ESN to address the aforementioned problem and improve the performance of ESN. The MI optimization method mainly consists of two parts: First, the scaling parameters of multiple inputs are adjusted based on the MI between the network inputs and outputs. Second, some output weight connections are pruned for optimization based on the MI between reservoir states. Furthermore, three MI-ESN models are proposed for a fed-batch penicillin fermentation process. Our experimental outcomes reveal that the obtained MI-ESN models outperform the ESN models without optimization and other traditional neural networks.

Energy consumption prediction of office buildings based on echo state networks

In this paper, energy consumption of an office building is predicted based on echo state networks (ESNs). Energy consumption of the office building is divided into consumptions from sockets, lights and air-conditioners, which are measured in each room of the office building by three ammeters installed inside, respectively. On the other hand, an office building generally consists of several types of rooms, i.e., office rooms, computer rooms, storage rooms, meeting rooms, etc., the energy consumption of which varies in accordance with different working routines in each type of rooms. In this paper, several novel reservoir topologies of ESNs are developed, the performance of ESNs with different reservoir topologies in predicting the energy consumption of rooms in the office building is compared, and the energy consumption of all the rooms in the office building is predicted with the developed topologies. Moreover, parameter sensitivity of ESNs with different reservoir topologies is analyzed. A case study shows that the developed simplified reservoir topologies are sufficient to achieve outstanding performance of ESNs in the prediction of building energy consumption.